跳转至

超越推理轨迹：将可解释的推理状态读出模块与原生 MoE 路由进行耦合

文章背景与核心概要

推理模型在最终输出中写下的内容，往往只是产生该结果的实际认知过程的局部记录。为了让潜在的过程状态既可读又具可操作性，本文针对混合专家（Mixture-of-Experts, MoE）推理模型引入了一种新颖的两级内部读出（two-level internal readout）框架。

该技术的核心首先在于潜空间蒸馏（\(J_{64}\)）：作者将词表规模的 \(J\) 空间蒸馏为 \(J_{64}\)——一个直接从模型推理状态中学习得到的 64 轴语义框架。这种读出方式揭示了隐藏的过程动态（例如将纯粹的推理努力与问题诱导的压力区分开来），并在预测任务中显著优于传统的词元占用（token-occupancy）基线。其次是低开销代理（\(R_{64}\)）：利用原生专家路由统计数据对 \(J_{64}\) 进行高效重建，从而生成 \(R_{64}\)。该代理在多个模型和架构族之间与 \(J_{64}\) 保持了高度相关性，同时保留了几乎全部的预测优势。最后，它支持了可操作的决策制定，包括测试时的候选选择与加权投票、基于滚动读出窗口的生成控制（停止并重新采样策略），以及通过直接修改路由机制来诱导预期的推理行为，成功将试错式的数值猜测转变为精确的符号执行。

📌 Summary

What a reasoning model writes in its final output is often only a partial record of the actual cognitive process that produced it. This paper introduces a novel two-level internal readout framework for mixture-of-experts (MoE) reasoning models to make latent process states both readable and actionable:

  1. Latent Distillation (\(J_{64}\)): The authors distill the vocabulary-scale \(J\)-space into \(J_{64}\), a 64-axis semantic frame learned directly from the model's reasoning states. This readout exposes hidden process dynamics—such as separating pure inference effort from problem-induced strain—and significantly outperforms token-occupancy baselines in predictive tasks.
  2. Low-Overhead Proxy (\(R_{64}\)): \(J_{64}\) is efficiently reconstructed using native expert-routing statistics to produce \(R_{64}\). This proxy achieves high correlation with \(J_{64}\) across models and architectural families while retaining almost all of the predictive advantages.
  3. Actionable Decision-Making:
  4. Test-time Selection: Improves candidate selection and weighted voting.
  5. Generation Control: Drives a cumulative stop-and-resample policy via rolling readout windows.
  6. Router Interventions: Direct modifications to the router mechanism successfully induce predicted reasoning behaviors, shifting stalls from trial-and-error numerical guessing toward precise symbolic execution.

Abstract

What a reasoning model writes is only a partial record of the process that produces it. We introduce a two-level internal readout for mixture-of-experts reasoning. We first distill vocabulary-scale J-space into J64, a 64-axis semantic frame learned from the model's own reasoning states. J64 reveals readable process state that the emitted trace does not show: it separates inference effort from problem-induced strain. It also adds 0.096 to 0.135 held-out AUC over a baseline that reads the same rollout as token occupancy and aggregates it in exactly the same way. We then reconstruct J64 from native expert-routing statistics. The result is R64, a low-overhead proxy: its median per-axis correlation with J64 is 0.69 to 0.86 across three models and two families, and on gpt-oss-20b it preserves 95 to 100% of J64's predictive gain. The readout supports test-time decisions at two temporal resolutions. Over completed candidate sets, J64 and R64 improve single-branch selection, and R64-weighted voting improves plain majority voting in seven of eight settings. During generation, rolling readout windows drive a cumulative stop-and-resample policy whose operating point is fixed on training questions alone. J64 improves accuracy by 1.1 to 5.9 points over a sibling-permuted control, and the routing-only R64 proxy retains 0.9 to 3.2 of those points. Finally, router edits aimed at the mechanism J64 names induce the predicted reasoning behaviors and shift a diagnosed stall from numerical guessing toward exact symbolic execution. Together, J64 makes latent process state readable, while routing makes it deployable and actionable.

摘要

推理模型所写下的内容只是产生该过程的局部记录。我们为混合专家（MoE）推理引入了一种两级内部读出机制。首先，我们将词表规模的 \(J\) 空间蒸馏为 \(J_{64}\)，这是一个从模型自身的推理状态中学习得到的 64 轴语义框架。\(J_{64}\) 揭示了输出轨迹所无法显示的、可读的进程状态：它将推理努力与问题引起的压力区分开来。与将相同的 rollout 读取为词元占用并以完全相同方式聚合的基线相比，它在保留集 AUC 上提升了 0.096 到 0.135。随后，我们通过原生的专家路由统计数据重建了 \(J_{64}\)。其结果是 \(R_{64}\)（一种低开销代理）：在三个模型和两个架构族中，它与 \(J_{64}\) 的每轴中位相关性达到了 0.69 到 0.86，并且在 gpt-oss-20b 上保留了 \(J_{64}\) 的 95% 至 100% 的预测增益。该读出机制在两个时间分辨率下支持测试时决策。在已完成的候选集上，\(J_{64}\) 和 \(R64\) 改善了单分支选择，\(R64\) 加权投票在八分之七的设置中改进了朴素的多数投票。在生成过程中，滚动读出窗口驱动了一种累积的停止并重新采样策略，其操作点仅固定在训练问题上。与同胞排列控制（sibling-permuted control）相比，\(J_{64}\) 将准确率提高了 1.1 到 5.9 个百分点，而纯路由的 \(R_{64}\) 代理保留了其中的 0.9 到 3.2 个百分点。最后，针对 \(J_{64}\) 所命名的机制进行的路由器编辑，成功诱导出了预测的推理行为，并将诊断出的停滞从数值猜测转变为精确的符号执行。总之，\(J_{64}\) 使潜在的过程状态可读，而路由使其可部署且可操作。

Key Highlights

  • \(J_{64}\) Semantic Frame: Uncovers hidden cognitive metrics (inference effort vs. strain) with substantial AUC improvements (0.096 to 0.135 gain over token-occupancy controls).
  • \(R_{64}\) Routing Proxy: Leverages native MoE routing metrics as a lightweight alternative, boasting a median per-axis correlation of \(0.69\) to \(0.86\).
  • Temporal Resolutions for Test-Time Decisions:
  • Completed Candidates: Enhances branch selection and weighted voting systems.
  • Active Generation: Powers dynamic stop-and-resample policies.
  • Targeted Router Edits: Directly addresses reasoning stalls by nudging the model away from empirical guessing and toward exact symbolic execution.

核心亮点

  • \(J_{64}\) 语义框架： 揭示了隐藏的认知指标（推理努力与压力），在保留集 AUC 上取得了显著提升（比词元占用控制高出 0.096 至 0.135）。
  • \(R_{64}\) 路由代理： 利用原生 MoE 路由指标作为轻量级替代方案，每轴中位相关性高达 \(0.69\) 至 \(0.86\)。
  • 用于测试时决策的时间分辨率：
  • 已完成的候选集： 增强了分支选择和加权投票系统。
  • 主动生成过程： 驱动动态的停止并重新采样策略。
  • 有针对性的路由器编辑： 通过引导模型远离经验猜测、转向精确的符号执行，直接解决推理停滞问题。

链接与资源